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§0. Introduction. 

Instanton vector bundles were defined by Atiyah, Drinfeld, Hitchin and Manin 
[ADHM] in order to construct all the self-dual solutions of the Yang-Mills equation 
over S*^. A mathematical instanton bundle £^ on P'^ :— P'^(C) can be defined as the 
cohomology bundle of a monad 

Oi-1)'' — > o2'^+2 — > 0{l f 

on P'^, where C2{E) — k. This is equivalent to the condition that £' is a stable 
bundle of rank 2 on P^ such that ci{E) = 0, C2{E) = k and H'^{E{-2)) =0. If £; 
is a mathematical instanton bundle, then it is easy to check by using Hirzebruch- 
Riemann-Roch Theorem that h^{S^E) - h^{S^E) = 8k-3 and so 8fc - 3 is the ex- 
pected dimension of the moduli space of mathematical instanton bundles MIp3{k). 
It is not known if the moduli space Ml^aik) is a regular variety of pure dimension 
8fc — 3. The answer is affirmative in the cases 1 < fc < 4 and this was proved in 
[H], [ES] and [LeP]. In this article we extend these results to the case k — 5. More 
precisely, we prove the following 

Theorem 0.1. For 2 < k < 5 the moduli space MIp3{k) of mathematical instan- 
tons is a regular variety of pure dimension 8k — 3. 

Our result should be compared with [A02], where it is proved that the closure of 
M/p3(5) in the Maruyama scheme of vector bundles of rank 2 with ci = 0, C2 = 5 
contains singular points. Our proof requires tools both from invariant theory and 
algebraic geometry. 

In §1 we prove some algebraic lemmas. In §2 we give an invariant theoretical 
description of MIp3{k) and we prove some result about unstable planes. Finally, 
in §3 we prove Theorem 0.1. 

The authors are grateful to A. N. Tyurin for fruitful advices. The first author 
thanks the Italian GNSAGA-INDAM for financial support. The second author is 
a member of the European network EAGER. 
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§1. An INVARIANT THEORETICAL DESCRIPTION OF M7p3(fc). 

Our first goal is to describe the moduli space MJp3(fc) in terms of invariant 
theory and prove that for any € Mlps^k) and any plane H in h^{EH) < 1- 
The group SL2fc+2 acts canonically on the space C^'^+^. Let /ii, /i2fc+2 be the 
standard basis of the space 0^*^+^ and h\, •••,/i2fe+2 be the dual basis of the dual 
space c(^'^+^)*. Consider the 2-form 

l<i<fe+l l<i<fc+l 

Let Sp2fc+2 be the stabilizer of the 2-form lo in the group SL2fe+2. The 2-form u 

defines canonically the Sp2fe_|_2-isomorphism (C(2fe+2)* ~ C^'^+^. We have the canon- 
ical actions of the group SL4 x SLfe x Sp2j;+2 the spaces C, C^* , C^*'+^, C'^, 
C^*, C4(8)C^.... 

We have the quadratic (SL4 x SLfc x Sp2fe_|_2)-niorphism 

7 : (g) C*^* (g) C^^+^ 5^C^* A^C*"*, 

(/; ® b* ®hif;,® b*, ® h,) ^ ^u;{hi, hi,) ifu;,) ® {b* A b*,) 

and the bilinear (SL4 x SL^ x Sp2fc_|_2)-niorphisms 

P{ri®b*®hu hv) = u{hi,hi,)f: ^b* 

and 

£ : C^* (g) C2'=+^ X C^^C'^ €^'=+2, 
if* ^bj^hi, fi' bj,) !-»■ 6u'6jfhi 

Consider the following conditions for an element A e C^* C'^* C'^'^'^'^ 
(El) e{A, / 6) ^ for all 7^ / e C^, 7^ 6 e C*^, 

{E3) P{A, /i) ^ for all 7^ /i e C^''+^. 

An element A G C^*0C'=*0C2'=+2 defines the sheaf morphism e»2fc+2 0{1)''. 
This morphism and the symplectic structure over C)2fe+2 (jeg^g ^j^g sequence 

(1.1) e)(-i)'= ^ c»2fc+2 ja^ o{i)''. 

The condition (iiJi) means that Ja is surjective or that Ker Ja is locally free. The 
condition {E2) means that the above sequence is a complex. Therefore, (Ei) and 
{E2) together mean that it is a monad according to [BH]. The condition (E^) means 
moreover that the cohomology bundle E of the monad is a stable vector bundle. It 
is well known (see e.g. [AOl], Theor. 2.8) that the conditions {Ei) and (£'2) imply 

Set 

Ii = {A€ C** C2*^+2 I the condition {Ei) holds for A}, 
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/:= 7i 0/20/3 = hnh. 

Recall that the quotient I/G, where G = SL^ x S-p2k+2 ^*^* is isomorphic to 
MIp3{k) (see [BH]). Moreover, the stabilizer of any ^ S / in the group G is equal 
to the kernel H of the action of G on / and dim(//) = 0. Therefore, any G-orbit 
G-A, where A e /, is isomorphic to G/// and dim(G- A) = dim(G) = 3P+5A;+3. 
Consider the canonical morphism 

n : I ^ I/G = M/p3(fc). 

The results above imply the following fact: 
Lemma 1.1. For any A € I we have 

dim(r^(A)M/p3(A;)) = dim(TA/) - Sfc^ - 5A; - 3. 

Theorem 1.2. Let E be an instanton bundle on and let H be a plane. Then 
h\E\H) < 1. 

Proof. The bundle E is the cohomology bundle of a monad 

where C2{E) = k. Notice that it is enough to show that h'^{K\^jj) < 1, where 
K := Ker6. A section s € H^{K\h) induces the diagram 


Oh ^ Oh 

S 

K\H Of+^ Onil)' 

T Of+^ Oh{1)'' 



Since i o s is injective, we see that s splits K\h as K\h — Oh © T. Therefore 
it is enough to show that H°{T) = 0. Now T is a bundle of rank A: + 1 with 
detT = 0{-k) and thus 

T~ A'=T*(-fc). 

We will prove a little more, indeed we show that 

(1.1) H°{A'T*{-i)) = for 1 <i < fc. 

We prove (1.1) by induction. For i = 1 (1.1) holds by the above diagram. Now, 
let us consider 

O^EiH -^Oh®T* ^ ©1(1) 0. 
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Lot us remember that, since ii is a rank 2 bundle with ci(E) = 0, we have A^E ~ 
O. Therefore, the second wedge power of T* twisted by Oi/(— 2) gives 

— > Oh{-2) — > T*(-2) e T*(-2) — ^ T*(-l)*^ 

which proves (1.1) for i = 2. Moreover, the i-th wedge power of T* twisted by 
0{—i) gives 

0—^ A'T*{-i)(BA'-^T*{-i)^ [A'-^T*{-i + l)f ®[A'-'^T*{-i + l)]'" 
and this sequence provides the inductive step. 

□ 

Definition. W{E) = {H € P^* | h°{E\H) + 0} is called the variety (scheme) of 
unstable planes of E. Its scheme structure is defined as the degeneracy locus of the 
mapping 

H^{E{-1)) eg) O ^ H\E) (g) 0(1) 
over P^* (Theorem 1.2 shows that this map drops rank at most by one). 
For an element ^ e C^* ® C*^* (g) C'^''+'^ define the subvariety 

Xa = {(AF) e P^* X P'^-^* I /* 6* G lm{/3{A, ■))}. 

Lemma 1.3. Let qi be the projection o/P^* x P'^-^* on P^*. We have W{E) = 
qi{XA) and the fiber of the projection Xa qi{XA) over H is isomorphic to 

Proof. With the notations of the proof of Theorem 1.2 we have that H G W{E) iff 
h^{K\H) ^ 0. We have H"{Kih) ^ Ker(C2'=+2 — ® C). Now suppose that 
corresponds to H, then the existence of a nonzero a G H^{K\h) is equivalent to 
P{A, a)=f*^b*, where (f*,¥) e P^* x P'^-i* . 

□ 

Corollary 1.4. The morphism Xa —>■ gi(X^) is an isomorphism of the underly- 
ing varieties, in particular dimX^ = dim(j'i(X^). □ 

Recall that special 't Hooft bundles are the instanton bundles such that h^{E{l)) = 

2. They can be defined through the Serre correspondence by fc + 1 skew lines lying 
on a smooth quadric surface [H]. We need the following special case of a theorem 
of J. Coanda [Co]. 

Theorem 1.5. If E is an instanton bundle such that dimW{E) > 2, then E is a 
special 't Hooft bundle and W{E) is a quadric surface. □ 

It is known ([H]) that special 't Hooft bundles are smooth points in the moduli 

space of instanton bundles. 

Corollary 1.6. If dim X a > 2, then A corresponds to a smooth point in the moduli 
space of mathematical instanton bundles. □ 

A reformulation of this Corollary into the invariant theoretical language is as 
follows. 
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Corollary 1.7. For any A° e / such that dim(X^o) > 2, we have 

dim(T^(^o)M7p3(fc)) = 8fc - 3. 

Lemma 1.8. Suppose S /; then dim(T4o/) > 3fc^ + 13fc if and only if there 
exists 7^ 5" e ^^C* (g) A^C'^ such that ^(A",S'°) = 0, where ^ is the bilinear 
SL4 X SLfc X Sp2k+2'''T^orphism, defined by 

^ : (g) C*^* (g) C=^'=+^ X S'^C^A^C'^ «) C*^ (g C2'=+^ 

(/* (g) 6* /i;, fi'fi"^bfAbj") !->■ {Sii'fi" +6i»ifi>)^{Sj'jbj" -6j"jbj>)(S)hi. 

Proof Wc have dim(rAo/) > 3fc2 + 13fc iff the differential ^7! j^a IS nonsur- 
jective. The differential d'y\AO is nonsurjective iff (^71^0)* is noninjective, i.e., 
(d7|Ao)*(5°) = for some 7^ 5° G S'^C^ A^C*^. It can be easily checked that 

id^\Ans)^aA,s). 

Hence, dim{TAoI) > 3k^ + 13k iff ^{A°,S°) = for some ^ S° € S^C^ (g) A^C*^. 

□ 

For the convenience of the reader we give a cohomological interpretation of 
Lemma 1.8. Let E'^ be the instanton bundle defined by ^4" e / as the cohomol- 
ogy bundle of monad (1.1). By Lemma 1.1 and deformation theory the assump- 
tion dim(T4o7) > Sfc^ + 13fc is equivalent to h'^{S'^E^) = dim(T„(^o)M7p3(fc)) > 
8k — 3. Therefore, the assumption of Lemma 1.8 is equivalent to H^{S^E^) ^ 0. 
The second symmetric power of the left hand side of (1.1) gives H'^{S'^E^) ~ 
i7^(S'^(Ker /a"))- The second symmetric power of the right hand side of (1.1) 
gives 

772(52(Ker /^o)) ~ Coker [H^{0{1)) (g (g <C'^''+^* H°{0{2)) (g A^{&*)] . 
Lemma 1.8 follows because the dual of $ can be identified with £,{A^, •). 

§2. Algebraic lemmas. 

Among all this section we prove some algebraic lemmas that we will use in order 
to prove our main result. 

Lemma 2.1. Suppose R is a nonzero block-matrix: 

where W is a skew- symmetric matrix of size k x k, then there exists a column vq 
of height k such that 

Rvo = ^0 
V A2M0 J 

for some column Uq of Jieight k, Ai, A2 € C. 

Proof. Suppose that det{R^) ^ 0. In this case set vq e Ker(ii2 — fioR^), where fiQ 
is a root of the equation det{R'^ — ^iR^) = 0. 
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Suppose that det{R ) = 0. One can assume that 

Di _ /^-Rii o\ u2 _ f Rli RI2 

^ " V oj' ^ -\El, Rl, 

where Rli is a skew-symmetric matrix of size k' x k', k' < k, det{Rli) ^ and 
Rii is a skew- symmetric matrix of size k' x k'. If R12 7^ or i?22 7^ 0, then we set 

vq = ( for some v'q such that i?i2i^o 7^ or i?22^o 7^ 0. If -Rf2 = and i?22 = 0, 
then i?2i = and we set vq = i ]■, where 







^0. 



□ 



Consider the linear spaces and C*^. Let /i,...,/4 be the standard basis of 
C and let /i , be the dual basis of the dual space C*. Let h\,...,hk be the 
standard basis of and let 6*, 6^ be the dual basis of the dual space C*^*. The 
group SL4 acts canonically on the space and the group SL/j acts canonically 
on the space C*'. So the actions of the group SL4 x SL^ are defined on the spaces 

Consider the linear space S'^C'* O A^C'^. For an element S € S'^C'^ O A^C'^ define 

rk(S) = dim(Im(p(5,.))), 

where 

iUU' A by,, f* (g) bj) ^ {5,,.,U' + ® {^j'jbj" - 5j"jby) 

is the bilinear (SL4 x SLfe)-morphism. Note that rk(S') is an even number. The 
following lemma is the only place in the paper where we need the assumption 
fc < 5. 

Lemma 2.2. Suppose 2 < k < 5 and consider S S S'^C (g) A^C^, such that 

2 < rk(5) < 2k — 2. Then one of the following conditions holds: 

(1) p{S,B*°) = /o ® 6V for some B*° € C^* ® C*^*, G C^, foO e 

(2) rk(5) = 6 and there exists 7^ /*° G C^* such that p{S, f*°^b*)=0 for 
all b* eC"*. 

(3) rk(S') = 8 and dim(Zs) > 2, where 

Zs = {(AF) eP3* X p'^-i* I p{S,f* ^b*) = 0}, 

p3* _ pi^^* p/c — 1* _ pi^^* 



Proof. Consider the coordinate expression of S in the bases {fi} and {bi}: 

S = cT%fifp®hiAbj. 
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We get a block matrix a defined by 



/ a 



12 



21 



„2fe 



/ 



where cr*^ = (o'lp)i<i.p<4 is a symmetric matrix of size 4x4, a'^^ = — cr-^*. There is 
a second coordinate expression 

S = ^Zfifj ®bi^ hp, 
and we get a second block matrix a defined by 

a = (S''-')i<ij<4 



where a*-' = i^\'!p)i<i,p<k is a skew-symmetric matrix of size k x k, a^^ = 
Transform the basis {bi} and obtain 



/a- 






524 


522 


523 




J32 


J33 


534 




542 


543 


544/ 



(2.1) 
We have 



det 1 / 12 \ 

r = rk(cr ) 



max {rk(ciifT*-'c2j)}. 

{(c»j)l<i,3<fe} 



2k -2 > rk(5) = rk(c7) = rk(5) > 2ik{a^'^) = 2r. 

Therefore, one of the following cases holds: 

(a) r = 1 or 2, 

(b) r = 3, rk((7) = 6, and k > 4, 

(c) r = 4, rk(f7) = 8, and k = 5, 

(d) r = 3, rk((T) = 8, and /c = 5. 

Transform the basis {/,} and obtain 



(2.2) 



^12 



1 if 1 <l=p<r, 

if l^porl=p>r. 



Prom (2.1) it follows that aj^ = for Z,p > r whence 
(2.3) a'^ =0 for i,j > r. 

(a). Consider the case (a). 

In this case we prove that the condition (1) holds, i.e., we prove that there exists 
a column /° of height 4 and columns 6°, B*°i, . . . , of height k such that 



^ 0. 
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But this easily follows from Lemma 2.1 and (2.3). 
(b). Consider the case (b). 

In this case we prove that the condition (2) holds, i.e. we prove that there exists 
a column /*° of height 4 such that 



(2.4) 



\btrj 



for any column b* of height k. Prom the condition rk(cr) = 6 and (2.1) it follows 
that 



(J J = 





tj 


IJ 
0-13 




ij 


ij 
^22 


ij 
'^23 





ij 


ij 
C^32 


ij 
^33 





V 








0/ 



Prom this for 



it easily follows (2.4) 

(c). Consider the case (c). 

In this case we prove that the condition (3) holds. We have: 



Zs = {(/*, b*) = ( 




\btr 



0}. 



Consider the matrix 



E4 
E4 



,13 



rl4 



where E4 is the identity matrix of size 4x4. The 8 rows of the matrix a are the 
first 8 rows of the matrix a. Since rk(0-) = 8 = rk(cr), for a matrix P of size 20 x p 
we have: 



(2.5) 



aP = 



iff 



aP = 0. 



Por 3 < i < 5 consider the following matrix Pj of size 20 x 4: 

a 

P^3 
P^i 
V fiS / 

where Pa = — E4 and = for j 7^ i. We see that a ■ Pi 
follows that a • Pj = or 



0. Prom (2.5) it 



a ■'a 



a ■'a , 



3 < j < 5. 
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Prom this we obtain 







a ■'a + [a a 
3 < i,j < 5. 



a a ) 



One can rewrite these equations into the following compact form: 
(2.6) [ha'^ + ^2^" + tsa'', + ha^^ + is^^^] = 

for all ti,t2,t3G C. 

Claim 1. For every {b^, 64, 65) ^ (0, 0, 0) there exists {bl, 62) o-nd a nonzero column 
/* of height 4 such that 

a : =0. 

\birj 

Proof of Claim 1. Prom (2.6) it follows that the symmetric matrices 



*„13 



bla 



O4CT + OgCr , 03(7 + 04(7 + OgCr 



commute therefore they have a common eigenvector /* with the eigenvalues 62) 
—b*i respectively. We have 

(hlf*' 



= 



and from this and (2.5) Claim 1 follows. 



□ 



From Claim 1 it follows that dim(Zs) > 2. 
(d). Consider the case (d). 

In this case we prove that the condition (3) holds, i. e., we prove that dim(Z5) > 

2. 

Claim 2. Suppose N c PC^* is a line in general position; then there exists ^ 
/*o e C^*, 6*0 G TV such that p{S, f*° O 6*°) = 0. 



Proof of Claim 2. One can assume that N = (61, 62)- We have to prove that there 
exists a column /*° of height 4 and 61, 62 G C, (6|, 63) ^ (0, 0) such that 



(2.7) 



b*2f*° 



V / 



= 0. 



Consider the 4*^* and 8*'* rows of the matrix a: 

row4(CT) = {0,...,0,allall...,all,all), 
row8((T) = {0,...,0,alf,all...,all,all). 
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Wc want to show that row4((j) and rowg (cr) are Unearly dependent. Suppose 
that row4((7) and rowg (cr) are Unearly independent, then the first 8 rows of the 
matrix a are linearly independent. Since rk(CT) = 8, we see that every row of u is a 
linear combination of the first 8 rows. From row4(CT) ^ it follows that cr]] ^ for 
some 3 < i < 5, 1 < j < 4. Since a'^h = —a-\) ^ 0, we see that (4(z — 1) + j)th row 



r0W4(i_i)+j((Tj - {crji,CFj2, Crj3> <^j4> Crj2, <JjZ, <Jji, ■■■) 

of the matrix a is not a linear combination of the first 8 rows. This contradiction 
proves that row4(o-) and vaws{(j) are linearly dependent. 
Finally, to obtain (2.7) we take 



/*° = 

and 6i, 62 such that (6^ , 62) ^ (0, 0) and bl row4(cr) + 62 rowg (cr) = 0. 

□ 

From Claim 2 it follows that dim(Zs) > 3 > 2. 

□ 

§3. The proof of Theorem 0.1. 

Lemma 3.1. Consider elements € C^* (g) C'^* C2'=+2 and 5° G S'^C^ (g) A^C'^ 
such thatC{A",S") =0. 

(1) Suppose 

T : C^* (g) &* (g) C^''+^ X S'^C^ A^C'^ X C'^''+'^ C^^& 
is an arbitrary trilinear (SL4 x SL^ x Sp2j,_|_2)-wor7)/iism; then 

t(A°,5°,C2'=+2)=0 

(2) Suppose 

is an arbitrary trilinear (SL4 x SL^ x Sp2k+2) -''TT'Orphism; then 

Proof. 

(1) Consider the following nontrivial trilinear SL4 x SL^ x Sp2fc_|_2"™orphism: 
To : (g) C'^* (g) €^'=+2 X S'^C^ (g A^C'^ x €^'=+2 ^ (g) C'', 
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where 

X : ® C*^* (g) €^'=+2 X C^C*^, 
{fi ® bj ®hi,hv) ^ u!{hi , hi>)fi ® bj 

is the bihnear SL4 x SL^ x Sp2fe_|_2"™orphisni. 

On the other hand the (SL4 x SL^ x Sp2fc_|_2) -module 

contains the irreducible (SL4 x SLk x Sp2/j+2)"™0'^ulc ® C*^ with multipUcity 
1. Therefore, there exists a unique, up to a scalar factor, nontrivial trilinear 
(SL4 X SLfe X Sp2fc_|_2)"™orphisni 

C"* (g) C*^* ® C2'=+2 X (S^C^ (g) A^C*^) X €^'^+2 C*(gC'=. 

Thus, T = CTo for some c G C and we get 

t(^o,50,C2'=+2) = cto(AO,50,C2'=+2) = cx(C(AO, 5°), €^^+2) = 0. 

(2) Consider the linear mapping a{A°,S'^, •)* dual to a{A^,S°, •) . From (1) it 
follows that Q!(A0, 50, •)* = 0- Thus, a(yl0, 5°, C^* C^*) = 0. 

□ 

Proof of Theorem 0.1. 

We suppose that there exists & I such that dim(T^(^o)M7p3(fc)) > 8fc — 3, 
2 <k <h and we obtain a contradiction. 

From Corollary 1.7 it follows that 

(3.1) dim(X^o) < 2 

and by Lemma 1.1 we have dim(T4o7) > 3fc^ + 18fc. Hence, by Lemma 1.8 there 
exists ^ 5° e ^^C^ ® A^C" such that 

(3.2) e(^°,5°)=o. 

Consider the following composition of linear mappings 

p{S°,-)°P{A°,-) : C2'=+2^C4®C^ h ^ p{S°,p{A°,h)), 

where /? is defined in §1 and p is defined in §2. From (3.2) and Lemma 1.8 (1) it 
follows that p{S°, •) o p{A°, •) = or 

(3.3) Im(/3(A0,.))cKer(p(5°,-))- 

On the other hand, by (E3) we have rk(/J(A°, •)) = 2A; + 2 and with (3.3) this gives 
us 

(3.4) rk(p(5°, •)) < 2fc - 2. 

Therefore, from (3.4) it follows that one of the conditions (1) - (3) of Lemma 2.2 
holds for S = S°. 
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I. Consider the case when the condition (1) of Lemma 2.2 holds for S = S^. 
In this case, consider the following composition of linear mappings 

£(A°,-)op(5°,-) : C^*^^* C2*=+2, B* ^ e{A°,p{S°,B*)). 

Prom Lemma 1.8 (2) it follows that e(A°, •) o •)) = 0. By the condition (1) 
of Lemma 2.2 there exists 5*^ G (g, C''* such that piS°, B*°) = /o (g) 6° ^ 0. 
Thus, we have s{A°,f O 6°) = s{A°,p{S°,B*°)) = and therefore A° ^ h. But 
this contradicts the fact that A^ e I. 

II. Consider the case when the condition (2) of Lemma 2.2 holds for S = 5*". 
Prom (3.4) it follows that k — 4 or k = 5. By the condition (2) of Lemma 2.2 

we have {/*°} x C''* C Kev{p{S°, •))• On the other hand, we have (3.3) and 

dim(Ker(p(S'°, •))) - dim(Im(/J(A°, ■))) = 

Therefore Im{P{A°, •) D {/*"} x M for some hnear subspace M c C*'* of dimension 
> 3. But this contradicts (3.1). 

III. Consider the case; whcin the condition (3) of Lemma 2.2 holds for 3 = 3^^. 
From (3.4) it follows that A; = 5. Thus 

dim(Im(/3(A°, •))) = 12 = dim(Ker(p(S'°, ■))) 

and from this, together with (3.3), it follows that Im(/3(A°,-)) = Ker(p(S'°, •))• 
Therefore X^o = Zgn. From this and the condition (3) of Lemma 2.2 we obtain 
dim(X^o) = dim(Zgo) > 2. But this again contradicts (3.1). 

□ 
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JO if fc = 4, 
1 2 if fc = 5. 
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